-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OCPBUGS-45290: Reject All CA-Signed Certs Using SHA1 #642
base: master
Are you sure you want to change the base?
Conversation
Skipping CI for Draft Pull Request. |
@gcs278: This pull request references Jira Issue OCPBUGS-45290, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
d2d10e7
to
36105f1
Compare
6b47be2
to
7bb99ea
Compare
/retest |
/jira refresh |
@gcs278: This pull request references Jira Issue OCPBUGS-45290, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
/jira refresh |
@gcs278: This pull request references Jira Issue OCPBUGS-45290, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
tested it with 4.18.0-0.ci.test-2024-12-06-060222-ci-ln-4l8tlmt-latest, same steps as the bug
/label qe-approved |
@gcs278: This pull request references Jira Issue OCPBUGS-45290, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
/retest-required |
Andy offered to help review (thanks!) |
// It's important that we do NOT reject self-signed certificates, as many root CAs still utilize SHA1. | ||
if !isSelfSignedCert(cert) { | ||
switch certs[0].SignatureAlgorithm { | ||
case x509.SHA1WithRSA, x509.ECDSAWithSHA1: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about DSAWithSHA1?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I originally left it out because DSA is unsupported by the router (DSA is deprecated), the router won't parse DSA certificate private keys:
router/pkg/router/routeapihelpers/validation.go
Lines 73 to 88 in 69f742b
switch t := key.(type) { | |
case *rsa.PrivateKey: | |
block = &pem.Block{ | |
Type: "RSA PRIVATE KEY", | |
Bytes: x509.MarshalPKCS1PrivateKey(t), | |
} | |
case *ecdsa.PrivateKey: | |
block = &pem.Block{ | |
Type: "EC PRIVATE KEY", | |
} | |
if block.Bytes, err = x509.MarshalECPrivateKey(t); err != nil { | |
return nil, err | |
} | |
default: | |
return nil, fmt.Errorf("block private key %T is not valid", key) | |
} |
So providing any DSA server certificate & key results in:
Invalid value: "redacted key data": block PRIVATE KEY is not valid
If I try to use a DSA intermediate cert, it get something similar, presumably because the x509 GO library doesn't support verifying DSA certificates:
[spec.tls.certificate: Invalid value: "redacted certificate data":
error verifying certificate: x509: certificate signed by unknown authority
(possibly because of "x509: cannot verify signature: insecure algorithm
DSA-SHA1" while trying to verify candidate authority certificate "www.exampleinter.com")
So, it looks like the router has always rejected certs using DSA keys. I can add a comment to document this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, so it is rejecting them properly so we know for sure a DSAWithSha1 (or any DSA) certificate doesn't break HAProxy router reload? Why not just add it here anyway?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, so it is rejecting them properly so we know for sure a DSAWithSha1 (or any DSA) certificate doesn't break HAProxy router reload?
Right.
Why not just add it here anyway?
Yea, it's a fair question. I originally left it out simply because it was already unsupported. But it's a fair point that adding it here would provide a more precise error message about DSA-SHA1, so that's a argument for adding it. I'll update.
I don't remember what this referred to and can't find the comment. What is the conclusion? |
69f742b
to
72201c5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @candita! Ready for another review.
// It's important that we do NOT reject self-signed certificates, as many root CAs still utilize SHA1. | ||
if !isSelfSignedCert(cert) { | ||
switch certs[0].SignatureAlgorithm { | ||
case x509.SHA1WithRSA, x509.ECDSAWithSHA1: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I originally left it out because DSA is unsupported by the router (DSA is deprecated), the router won't parse DSA certificate private keys:
router/pkg/router/routeapihelpers/validation.go
Lines 73 to 88 in 69f742b
switch t := key.(type) { | |
case *rsa.PrivateKey: | |
block = &pem.Block{ | |
Type: "RSA PRIVATE KEY", | |
Bytes: x509.MarshalPKCS1PrivateKey(t), | |
} | |
case *ecdsa.PrivateKey: | |
block = &pem.Block{ | |
Type: "EC PRIVATE KEY", | |
} | |
if block.Bytes, err = x509.MarshalECPrivateKey(t); err != nil { | |
return nil, err | |
} | |
default: | |
return nil, fmt.Errorf("block private key %T is not valid", key) | |
} |
So providing any DSA server certificate & key results in:
Invalid value: "redacted key data": block PRIVATE KEY is not valid
If I try to use a DSA intermediate cert, it get something similar, presumably because the x509 GO library doesn't support verifying DSA certificates:
[spec.tls.certificate: Invalid value: "redacted certificate data":
error verifying certificate: x509: certificate signed by unknown authority
(possibly because of "x509: cannot verify signature: insecure algorithm
DSA-SHA1" while trying to verify candidate authority certificate "www.exampleinter.com")
So, it looks like the router has always rejected certs using DSA keys. I can add a comment to document this.
That thread is the first thread when I review the changes: https://github.com/openshift/router/pull/642/files#r1881050540. Does that link work for you? |
/retest |
This is a suspicious error to see on this PR. Or is it coincidentally related to signing?
/test e2e-aws-serial |
It does look funny, but according to search.CI, I see a lot of other instances of this failure for other PRs, so I think it's not related. |
The issue with e2e-aws-serial appears to be a docker build failure:
/test e2e-aws-serial Cluster installation failures that don't look specific to ingress: |
// Unsupported destinationCACertificates algorithms won't prevent HAProxy from starting. | ||
// However, HAProxy will quietly refuse to use them at runtime. Rejecting here improves UX. | ||
if err := validateCertSignatureAlgorithms(certs); err != nil { | ||
result = append(result, field.Invalid(tlsFieldPath.Child("destinationCACertificate"), "redacted ca certificate data", err.Error())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not add the SHA1 issue to the error message? Otherwise the user still won't know what's wrong with the CA certificate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is already added to the error message via the err
return value from validateCertSignatureAlgorithms
x509.SHA384WithRSAPSS, | ||
x509.SHA512WithRSAPSS: | ||
return x509.RSA | ||
case x509.DSAWithSHA1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it important to include DSA here and not in validateCertSignatureAlgorithms
? If it wasn't included here I wouldn't have noticed it was missing in validateCertSignatureAlgorithms
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DSA was included here as a design choice because the function's responsibility is intended to be generic: signatureAlgorithmToPublicKeyAlgorithm
, it's simply converting the known signature algorithms to a public key algorithm type, similar to a function you'd find a library. It doesn't make any assumptions or limitations about whether DSA is supported or not, which (in my opinion) makes this code safer to future reuse.
I've since added DSA SHA1 to the rejection list, so it's now we use it in both functions you referenced.
switch cert.SignatureAlgorithm { | ||
// DSAWithSHA1 (as well as any DSA key algorithm) is already unsupported and rejected. | ||
case x509.SHA1WithRSA, x509.ECDSAWithSHA1: | ||
return fmt.Errorf("router does not support CA-signed certs using SHA1") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not collect/append the errors on the certs and return them all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah sorry I think I missed this comment in an earlier review.
Yea I can update it. It's an edge case that multiple non-self-signed certs using SHA1 or MD5, but it's possible nonetheless, and the function should handle it.
Still seeing this issue. Want to report it to TRT?
|
/test e2e-aws-serial |
Previously, only SHA1 leaf certs were rejected. However, in 4.16, any SHA1 cert that is CA-signed (not self-signed) is unsupported. This led to cases were routes with SHA1 intermediate CA certs were accepted, but HAProxy rejects them. Self-signed SHA1 certificates (i.e. root CA) remain supported since they are not subject to verification. This update ensures all route certs, including the server, CA, and destination CA certs, are inspected, and any SHA1 cert that is not self-signed is rejected.
72201c5
to
fb0668a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for another review @candita, comments posted
// Unsupported destinationCACertificates algorithms won't prevent HAProxy from starting. | ||
// However, HAProxy will quietly refuse to use them at runtime. Rejecting here improves UX. | ||
if err := validateCertSignatureAlgorithms(certs); err != nil { | ||
result = append(result, field.Invalid(tlsFieldPath.Child("destinationCACertificate"), "redacted ca certificate data", err.Error())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is already added to the error message via the err
return value from validateCertSignatureAlgorithms
// It's important that we do NOT reject self-signed certificates, as many root CAs still utilize SHA1. | ||
if !isSelfSignedCert(cert) { | ||
switch certs[0].SignatureAlgorithm { | ||
case x509.SHA1WithRSA, x509.ECDSAWithSHA1: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, so it is rejecting them properly so we know for sure a DSAWithSha1 (or any DSA) certificate doesn't break HAProxy router reload?
Right.
Why not just add it here anyway?
Yea, it's a fair question. I originally left it out simply because it was already unsupported. But it's a fair point that adding it here would provide a more precise error message about DSA-SHA1, so that's a argument for adding it. I'll update.
x509.SHA384WithRSAPSS, | ||
x509.SHA512WithRSAPSS: | ||
return x509.RSA | ||
case x509.DSAWithSHA1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DSA was included here as a design choice because the function's responsibility is intended to be generic: signatureAlgorithmToPublicKeyAlgorithm
, it's simply converting the known signature algorithms to a public key algorithm type, similar to a function you'd find a library. It doesn't make any assumptions or limitations about whether DSA is supported or not, which (in my opinion) makes this code safer to future reuse.
I've since added DSA SHA1 to the rejection list, so it's now we use it in both functions you referenced.
@@ -205,6 +205,11 @@ func ExtendedValidateRoute(route *routev1.Route) field.ErrorList { | |||
} else { | |||
tlsConfig.CACertificate = string(data) | |||
} | |||
// HAProxy will fail to start if intermediate CA certs use unsupported signature algorithms. | |||
// However, root CAs can still use unsupported algorithms since they are self-signed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are only checking the CACertificate here, and I think I was wondering about the other certs. So we specifically do not check the other certs in the tlsConfig variable for the reasons above.
Sorry, I'm a bit confused, we do check spec.tls.certificate
here:
if err := validateCertSignatureAlgorithms(certs); err != nil { |
And we check spec.tls.destinationCACertificate
here:
if err := validateCertSignatureAlgorithms(certs); err != nil { |
So we check every certificate provided equally (server cert, CACerts, and destinationCACerts), any certificate that HAProxy uses is subject the SHA1 rejection. Every type of cert gets the same treatment (e.g. self-signed Certs are exempt from SHA1 check).
Are we already doing point 2 or not?
We are currently doing point 2, by that I mean today we are unnecessarily forcing users to recreate/upgrade their self-signed SHA1 server certs in 4.15 before upgrading, when really HAProxy in 4.16 still allows/supports it (this PR will fix that).
switch cert.SignatureAlgorithm { | ||
// DSAWithSHA1 (as well as any DSA key algorithm) is already unsupported and rejected. | ||
case x509.SHA1WithRSA, x509.ECDSAWithSHA1: | ||
return fmt.Errorf("router does not support CA-signed certs using SHA1") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah sorry I think I missed this comment in an earlier review.
Yea I can update it. It's an edge case that multiple non-self-signed certs using SHA1 or MD5, but it's possible nonetheless, and the function should handle it.
I see someone posted about it slack and it seems like the image-registry team is aware of it and a recommendation has been made to fix it. I'll keep an eye on the progress. |
@gcs278: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Previously, only SHA1 leaf certs were rejected. However, in 4.16, any SHA1 cert that is CA-signed (not self-signed) is unsupported. This led to cases were routes with SHA1 intermediate CA certs were accepted, but HAProxy rejects them. Self-signed SHA1 certificates (i.e. root CA) remain supported since they are not subject to verification.
This update ensures all route certs, including the server, CA, and destination CA certs, are inspected, and any SHA1 cert that is not self-signed is rejected.